Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning

Offline reinforcement learning—learning a policy from batch of data—is known to be hard for general MDPs. These results motivate the need look at specific classes MDPs where offline learning might feasible. In this work, we explore restricted class obtain guarantees learning. The key property, which call Action Impact Regularity (AIR), is that actions primarily impact part state (an endogenous component) and have limited on remaining exogenous component). AIR strong assumption, but it nonetheless holds in number real-world domains including financial markets. We discuss algorithms exploit provide theoretical analysis an algorithm based Fitted-Q Iteration. Finally, demonstrate outperforms existing across different data collection policies simulated real world environments regularity holds.